Skip to content

to-dev: fix(webrtc): reuse existing Receiver when remote switches payload type mid-session - #2375

Open
misecichin wants to merge 1 commit into
AlexxIT:devfrom
misecichin:fix-webrtc-payload-type-switch
Open

to-dev: fix(webrtc): reuse existing Receiver when remote switches payload type mid-session#2375
misecichin wants to merge 1 commit into
AlexxIT:devfrom
misecichin:fix-webrtc-payload-type-switch

Conversation

@misecichin

Copy link
Copy Markdown

Bug

Some WebRTC producers start sending media on a different RTP payload type than the one negotiated at connection setup, without a full SDP renegotiation. I hit this consistently with a Nest doorbell (nest: source, protocols=WEB_RTC).

Conn.GetTrack (pkg/webrtc/producer.go) deduplicates Receivers by exact *Codec pointer identity:

for _, track := range c.Receivers {
    if track.Codec == codec {
        return track, nil
    }
}

When the remote peer switches payload type for the same Media, getMediaCodec resolves a different *Codec pointer (matching the new payload type), so this lookup misses and a brand-new, unwired Receiver is created — permanently starving every already-connected consumer (RTSP/WebRTC senders) of video, even though the producer is actively receiving valid frames on the new payload type.

Reproduction

go2rtc negotiates H264 on payload type 96 with the Nest media server. A few seconds into the session, Nest begins transmitting on payload type 98 instead (same media, no renegotiation). Before this fix, /api/streams shows:

"receivers": [
  {"id": 3, "codec": {"codec_name": "h264"}, "childs": [4]},                    // wired to the RTSP sender, 0 bytes forever
  {"id": 7, "codec": {"codec_name": "h264"}, "bytes": 787424, "packets": 730}   // orphan, no consumer, real data
]

ffmpeg consuming the RTSP restream fails with:

Could not find codec parameters for stream 0 (Video: h264, none): unspecified size

I reproduced this identically on v1.9.0 and v1.9.10, and also with an unmodified alexxit/go2rtc:latest — so it isn't a version-specific regression, and adding explicit video=h264&audio=opus hints to the source URL doesn't change the behavior either.

Fix

Reuse the existing Receiver for the same Media when no exact codec-pointer match is found, instead of creating an orphan:

case core.ModePassiveProducer, core.ModeActiveProducer:
    for _, track := range c.Receivers {
        if track.Media == media {
            return track, nil
        }
    }

This is safe because Receiver.Input just forwards raw RTP payload bytes to its children — it doesn't depend on which declared payload type the packet arrived on, and outgoing payload-type stamping is handled separately by the Sender side. I don't see simulcast/multi-encoding selection logic in this producer path that this would conflict with, but happy to adjust if I'm missing a case this affects.

Verification

After the fix, /api/streams shows a single Receiver correctly wired to the consumer, and ffmpeg reports a fully valid stream:

Stream #0:0: Video: h264 (High), yuv420p(progressive), 384x512, 30 fps, 15 tbr, 90k tbn
Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp

15MB+ of real video flowing through in a 12s test, versus 0 bytes on the wired receiver before the fix. Also validated against a real Frigate deployment (ffmpeg 7.0) pulling the RTSP restream in production, stable with no drops over multiple minutes, versus crash-looping every ~20s before.

…e mid-session

Some WebRTC producers (observed with the Nest/SDM source) start sending
media on a different payload type than the one negotiated at connection
setup, without a full renegotiation. Since Conn.GetTrack deduplicates
Receivers by exact *Codec pointer identity, a payload-type change for
the same Media creates a brand-new, unwired Receiver instead of reusing
the existing one - permanently starving every already-connected
consumer (RTSP/WebRTC senders) of video, even though the producer is
actively receiving valid frames.

Reproduced against a live Nest doorbell (WEB_RTC protocol): go2rtc
negotiates H264 on payload type 96, then the Nest media server begins
transmitting on payload type 98 a few seconds into the session. Before
this fix, /api/streams shows two H264 receivers - the wired one stuck
at 0 bytes forever, and an orphan one accumulating real data with no
consumer attached. ffmpeg consuming the RTSP restream fails with:

  Could not find codec parameters for stream 0 (Video: h264, none): unspecified size

After this fix, there is a single Receiver correctly wired to the
consumer, and ffmpeg reports a valid stream (H264 High profile,
384x512 @ 30fps) with megabytes of real video flowing through.

Reproduced and fix verified on v1.9.0 and v1.9.10; the bug is present
in both, so this isn't a version-specific regression.
@misecichin

Copy link
Copy Markdown
Author

Current docker with the fix which i'm using now for the Nest Doorbell 2:
misecichin/go2rtc-nest:1.9.10-nestfix

@ajplotkin

Copy link
Copy Markdown

Independent confirmation, from a different deployment and a different consumer path: this is real, and your diagnosis matches mine exactly.

I hit the same thing on a Nest Doorbell (wired) via nest: on go2rtc 1.9.14, with an RTSP consumer rather than Frigate. Same mechanism — Conn.GetTrack dedupes by *Codec pointer, Nest starts transmitting on a payload type that resolves to a different *Codec, the lookup misses, and a second unwired Receiver is created while every already-attached consumer starves. /api/streams shows two video receivers, one with childs and no bytes, one with bytes and no children.

What made it maddening to diagnose is that it is dial-order dependent: a cold dial mis-binds essentially every time, while a consumer attaching to an already-warm producer binds correctly. So it presents as "recording works if I open the camera first," which sends you looking in entirely the wrong place.

I filed it as #2389 before finding this PR — happy for that to be closed as a duplicate if you'd rather consolidate. #2386 is a third independent report (Nest Doorbell Gen 2 via Frigate, also stuck in a restart loop), and #2365 is arguably a fourth party hitting a downstream symptom of the same starvation. Cross-linking them here so it is clear this is not a one-off.

One suggestion on the patch itself

The lookup matches on Media alone:

for _, track := range c.Receivers {
    if track.Media == media {
        return track, nil
    }
}

core.Media.Codecs is a list, and getMediaCodec (pkg/webrtc/conn.go) selects among those codecs by the transmitted payload type. So "same Media" does not imply "compatible codec" — a video m-line that offered H264 and VP8 is ordinary WebRTC, and a remote moving from the H264 PT to the VP8 PT would, with this patch, be handed the receiver still described as H264. Consumers would then get VP8 RTP labelled H264, which fails in a much more confusing way than the bug being fixed.

Adding the compatibility check costs nothing and bounds it to the case you actually observed:

for _, track := range c.Receivers {
    if track.Media == media && track.Codec.Match(codec) {
        return track, nil
    }
}

Codec.Match compares name, clock rate and channels (and ignores FmtpLine), so a genuine PT-switch-within-the-same-codec still matches — which is the Nest behaviour — while a switch to a different codec falls through to the existing create path rather than being silently mis-wired.

The variant I have been running in production

For what it is worth, mine updates the receiver's codec rather than returning it as-is, and is gated to the Nest source to bound the blast radius:

if c.FormatName == "nest/webrtc" {
    for _, recv := range c.Receivers {
        if recv.Media == media && recv.Codec != codec && recv.Codec.Match(codec) {
            recv.Codec = codec
            break
        }
    }
}
track, err := c.GetTrack(media, codec)

Updating recv.Codec keeps the receiver's description in step with what is actually arriving, which matters for anything reading the codec downstream (the RTSP server copies FmtpLine into DESCRIBE). Your version is simpler and covers more sources, which I think is the better shape for upstream — I mention mine only because the two differ in that one respect and it may be worth a deliberate choice rather than an accident.

Running since 2026-07-31 across four Nest cameras: cold restarts now produce one video receiver with bytes flowing, zero starved receivers, and the "recording delivered audio but no video" failures are gone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants